System and method to measure effectiveness of business learning

ABSTRACT

The invention generally describes learning solutions and related methods for managing the learning and training needs of an organization in a manner that is responsive to dynamic business timelines, goals and strategic objectives. Business goals are determined through examination of existing business data sources, so as to tailor learning solutions to business impacts. Accurate computations of return on training investment are provided to allow accurate assessment of the effectiveness of training programs. Also disclosed are networks of learning platforms that together implement the learning solution by providing electronic tools and information sharing capabilities needed by a learning solution services provider to efficiently implement and manage learning efforts according to the business desires of a complex learner organization, while receiving organization performance data in order to compute and report performance measurements.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation-in-part of: U.S. patent applicationSer. No. 10/265,156, filed on Oct. 7, 2002, which claims the benefit ofU.S. provisional patent application Ser. No. 60/391,932, filed on Jun.28, 2002, and U.S. provisional patent application Ser. No. 60/391,929,filed on Jun. 28, 2002, which is hereby incorporated by reference in itsentirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable

NAMES OF PARTIES TO A JOINT RESEARCH AGREEMENT

Not Applicable

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not Applicable

BACKGROUND

1. Field of the Invention

The present invention relates to a learning solution and related methodsfor managing the learning and training needs of an organization in amanner that is responsive to dynamic business needs. In particular, thepresent invention relates to learning solutions that are designed inaccord with an organization's existing business data sources and goalsset relative to existing business data.

2. Background Art

Employee training and education are becoming increasingly more criticalto the success of organizations within today's modern global economy.Decision support capabilities are needed that ensure that ongoinglearning efforts are aligned with the business strategies and budgetpriorities as measured by quantifiable performance metrics. Suchdecision support capabilities must effectively collect and present todecision makers key indicators upon which to base their decisions.

Further, contemporary learning solutions fail to provide the ability forexecutives to monitor performance of learning solution efforts inachieving their underlying purpose of furthering business strategies.The inability to accurately monitor performance metrics leavesorganizations and their learning services with an inability to ensureservice quality, allow for accurate resource forecasts, allocate costsand provide a means to measure the learning solution's success helpingthe organizations meet goals and commitments. In contemporary learningsolutions, any performance data that is gathered is typically obtainedthrough surveys and self-reporting. Thus, even though modernorganizations track myriad types of data themselves, that actual data isnot used to measure the success of learning solutions.

Thus, there remains a need in the art for improvements in institutionallearning solutions and supporting processes.

SUMMARY OF THE INVENTION

In light of the above-described and other deficiencies inherent incontemporary learning solutions, the various embodiments of the presentinvention provide learning solutions and related methods that ensurethat an organization's instructional content and delivery mechanisms arealigned with the business and learning objectives of that organization.Such learning solutions and related methods are capable of dramaticallyimproving an organization's efficiency by identifying, prioritizing,developing and delivering employee training efforts and by providingdynamic system monitoring capabilities that ensure the effectiveness ofthe critical employee training efforts. Various aspects of the inventionprovide, among other things, tracking of performance metrics tobusinesses by leveraging existing data collection systems in order todesign learning solutions.

In this regard, a first aspect of the invention includes a method foraligning employee learning efforts with strategic business goals andpriorities of an employer organization. The method comprises receiving aset of business goals from the employer organization, and identifyingperformance data collected by the employer organization. The methodfurther includes transforming the business goals of the employerorganization into performance metrics relative to the, performance data,and designating business tracts and initiatives for use in sortingpotential learning efforts of the employer organization. These tractsand initiatives correlate to the performance metrics. Additionally, themethod includes receiving requests from the employer organization forlearning efforts to train the employees and sorting the requests intotracts or initiatives, and selecting appropriate learning effortapproaches for each received request. The appropriate learning effortapproaches are subject to a budget allocation for the tract orinitiative into which each learning effort request is sorted. Also, themethod includes monitoring the performance data subsequent to deliveryof the learning effort and computing changes in the performance datacaused by delivery of the learning effort, and reporting measurement ofthe performance metrics based on the computed changes in the performancedata.

A second aspect of the invention includes a network-based system toprovide coordination and synchronization of learning contentdevelopment, delivery, and management. The system includes a contentdevelopment module that itself includes submodules for receiving inputrelated to learning content and generating computer-deliverable learningcontent, and an electronic delivery module that itself includessubmodules for delivering instruction to students. The delivering ofinstruction in turn includes asynchronous computer-delivered instructionand synchronous virtual classroom instruction, with thecomputer-delivered instruction incorporating the learning content. Thesystem further includes a data collection module that itself includessubmodules for receiving performance data from an external data sourceand measuring the performance data against stored performance metrics.The performance metrics measure the results of the delivery of theinstruction to students. Additionally, the system includes a resourcessynchronization module that itself includes submodules for schedulingand allocating the utilization of learning resources in the delivery ofthe instruction and in the development of the learning content.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are included to provide further understandingof the invention and are incorporated in and constitute a part of thisspecification. The accompanying drawings illustrate embodiments of theinvention and together with the description serve to explain theprinciples of the invention. In the figures:

FIG. 1 is a schematic diagram depicting the interaction of the basicmodules of a learning solution according to embodiments of the presentinvention;

FIG. 2A is a flow diagram depicting a process utilized in embodiments ofthe present invention for alignment of learning efforts with prioritizedbusiness strategies;

FIG. 2B is a diagram depicting an exemplary learning blueprint thatcould be utilized in preferred embodiments of the invention toprioritize and fund learning efforts implemented by learning solutionsof the invention;

FIGS. 2C and 2D are diagrams depicting the mapping of plans for learningefforts onto a learning blueprint according to preferred embodiments ofthe present invention;

FIG. 3 is a flow diagram depicting a process for utilizing existingsources of data for measuring performance of a learning solution;

FIG. 4 is a flow diagram depicting an iterative process for developinglearning content according to embodiments of the present invention;

FIG. 5 is a schematic diagram depicting the interaction of deliverysub-modules of a learning solution according to embodiments of thepresent invention; and

FIG. 6 is a schematic diagram depicting a network of learning platformsthat communicate to provide coordination and synchronization of learningcontent development, delivery and management so as to enhance thelearning solution's response time to instructor, student, and/orresource concerns and issues.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

As used hereafter, the term “organization” is used broadly to identifythe business entity that is in need of a solution to provide learningservices. The learning solutions according to the present invention caneither be outsourced by such organizations (whereby the processes of thelearning solution are implemented by a third-party learning servicesprovider under contract to an organization) or can be implementedinternally within the organization (as, for example, an independentbusiness unit within the “outsourcing” organization). In embodiments ofthe invention wherein a learning services business unit within anorganization provides learning services to other business units withinthe organization, the learning services business unit would still feelthe need to satisfy customers as it would be responsible for generatinga “profit” by meeting predefined performance metrics set by “client”business units (i.e., the other more traditional business units of theorganization) and allocating learning effort charges as expenses to theclient business units as appropriate. In this manner, the learningsolutions according to the present invention can said to be“outsourceable” by an organization.

A learning solution 100 is conceptually depicted in the schematicdiagram of FIG. 1. Solution 100 includes modules of interrelated andinterdependent business processes, which may be automated by electronictools provided by various communication platforms as will be describedin more detail below, that manage and coordinate the learning solutiontasks of the prioritization of business learning needs and selection oflearning effort approaches, the development of learning content, thedelivery of learning content and instruction, and the administration oflearning solution performance. The interaction of the various modules ofbusiness processes with one another helps to define the services,performance metrics and application capabilities, among other things,required to support an optimized outsourceable learning solution to meeta given organization's business learning objectives.

As depicted in FIG. 1, business strategy and measurement alignmentmodule 110 includes processes that identify business strategies of theorganization and accept requests for learning efforts (e.g., employeetraining in a particular product line) from the organization (and/or itsbusiness units). It identifies performance data collected by theorganization against which success of a learning effort could bemeasured. It then aligns the organization's overall learning solutiongoals with its business goals by determining learning objectives of thelearning solution in terms of the collected performance data. Theprocesses of the business strategy alignment module thereafter identifypotential learning efforts to meet the identified learning objectives ofthe organization, and support the downstream development and delivery ofa tailored learning curriculum by providing relevant information toprocesses in the various other modules (120, 130, 140, and 150) ofsolution 100. This alignment ensures that learning curricula aredesigned to meet the organization's most current needs.

Once solution 100 identifies what types of learning or curriculum aredesired and/or required by the organization and prioritizes them inorder to meet business objectives, the solution is also able to deliversuitable learning materials to fill those needs. Content managementmodule 120 includes processes and associated tools required to buildand/or buy appropriate learning content based upon learning objectivesand delivery requirements. On one hand, it may be easier and cheaper tobuy precompiled learning content from a third-party as opposed todeveloping or compiling new content. Alternatively, the specializedneeds of the organization may make it difficult to utilize outsourced orprecompiled learning or training content. Content management module 120facilitates solution 100 taking factors such as these appropriately intoconsideration when assembling suitable learning content.

After learning content has either been obtained from a third-party ordeveloped internally by the learning solution, the business processes ofcontent management module 120 also create, manage and operate a centralrepository of learning content and knowledge objects. The learningcontent and knowledge objects are appropriately formatted, tagged andcataloged such that they can be reused as appropriate and may be easilyaccessed for delivery to the learner in the desired manner whenscheduled or requested.

Delivery module 130 in turn contains business processes and relatedtools for providing the delivery of high quality training across aplurality of synchronous and asynchronous media in a cost efficientmanner. The delivery business processes in module 130 facilitate, forexample, instructor led training, virtual instructor led training (suchas with a live instructor connected to students throughvideoconferencing or over a computer network like the Internet)including virtual classrooms, and interactive computersoftware-delivered training systems for self-paced learning. Further,the delivery business processes in module 130 manage coordinating thescheduling of the facilities where training takes place with appropriateinstructor and learner commitments.

Still referring to FIG. 1, learning administration module 140 containsbusiness processes and associated tools necessary to provide learningmanagement and administrative services across the entire learningsolution 100. Exemplary learning administration business processesinclude individual curriculum management, course registration, coursefulfillment management, and record keeping of training results.

Data collection module 150 contains business processes and associatedtools necessary to provide an interface between the organization's datasources and the learning solution 100. The data collection module 150assesses the organization's underlying data collection infrastructureand determines an appropriate mechanism for receiving data required toassess achievement of the learning objectives. This mechanism may beserver push or pull, and may use real-time or periodic updates,depending on the requirements of the organization, security concerns,network infrastructure, as well as other issues. Underlying data formatsused by the organization may affect the actual data fields received bythe data collection module 150. The data collection module 150 maycommunicate with the business strategy and measurement alignment module110 in order to revise learning objectives as necessary to accommodatethe actual data fields received. The data collection module 150 alsoincludes reporting capability in order to report on the data collectedfrom the organization.

Preferably, the learning solution 100 also includes a businessmanagement module (not depicted in FIG. 1) operating in communicationwith modules 110-150 and facilitating the overall operation of thelearning solution with and/or within the learner organization. Thebusiness management module includes processes and mechanisms that managethe overall lifecycle of a learning initiatives and efforts and providecontinuous improvement to currently ongoing and future learninginitiatives and efforts. The business management module assesses thesolution's compliance with overall strategic initiatives and businesspriorities of the learner organization by reviewing performance metricsthat are tracked and compiled by the data collection module, and alsooptionally uses those performance metrics to handle financialsettlements regarding the rendering of learning services to variousentities. The business management module thereby provides the learningsolution with the capability to manage the overall lifecycle of learninginitiatives and efforts and to provide continuous improvement tocurrently ongoing and future learning initiatives and efforts in a“profit-driven” manner.

The processes of the business strategy and measurement alignment module110 are central in ensuring that the learning efforts produced by thelearning solution entities are completely aligned with the strategicobjectives of the organization as expressed by the learning objectives.The business strategy and measurement alignment module 110 passes therequests from the organization (or business units thereof) for newlearning efforts or for modified learning efforts through a formalprocess whereby the requested learning effort becomes cast in aconsistent direction with organizational strategic objectives andpriorities, funded from appropriate budget allocations, and transformedinto a work in progress by the learning solution 110. Referring now toFIG. 2A, a business strategy alignment process 200 utilized in certainembodiments of the present invention is depicted and includes varioussteps that enable newly requested and ongoing learning efforts to bealigned with prioritized business strategies and goals.

As depicted in FIG. 2A, in step S210 business strategy alignment process200 first prepares a customized learning blueprint and businessmeasurement scope document for the organization. A learning blueprintprovides a grouping and organization of various business strategies andgoals across the entire organization and identifies potential areas oflearning. Each organization's learning blueprint is therefore unique andis used by the business strategy alignment process 200 and the businessstrategy alignment module to determine which potential learning efforts(from the many requested by various entities within the organization)introductions or modifications are funded (including from which budgetallocation and to what amount) and pursued (i.e., if the requestedlearning effort consistent with one or more stated learning strategiesor objectives reflected in the blueprint) and exactly how the request isfulfilled (i.e., requested learning efforts that map to higherorganizational priorities may be entitled to different funding orresources). An organization's learning blueprint is typically reviewedand revised periodically by the business strategy alignment module forthe organization (typically in light of information provided by thebusiness management module), such as once or twice a fiscal year toreflect changing business strategies and learning services budgets orinsights gained from business measurement studies. Of course, anorganization's learning blueprint can be reviewed and revised at anytime whenever significant changes in the learning focus and businessstrategies or priorities of the learner organization occur.

FIG. 2B depicts an exemplary learning blueprint 211 that could beutilized to prioritize and allocate funding to learning efforts andbusiness measurement studies implemented by learning solutions of theinvention for a particular hypothetical organization that is outsourcingits learning needs. As shown in FIG. 2B, a learning blueprint 211itemizes different strategic objectives of the organization into varioustracts 212 which can represent, for example, organizational areas offocus, product groups, business units, or departments within theorganization. For a given organization, as depicted in learningblueprint 211, a tract can be assigned, for example, for productsegments, sales skills, general employee development and organizationculture transformation. For each such tract, the learning blueprintitemizes initiatives 213 that reflect particular areas in which theorganization expects to focus a significant portion of its attention andexpenditures on learning efforts. Thus, an employee development tractcan contain separate initiatives for new employee orientation, andtraining regarding an upcoming organization-wide computer system change.Similarly, the product segment tracts can include separate initiativesfor major product lines (e.g., custom database solutions) or evenindividual products of high priority (e.g., new products introducing newproduct lines). For each tract 212 and/or each initiative 213 itemizedwithin the learning blueprint 211, overall budget caps can be assignedto assist the learning solution in obtaining, managing and deliveringlearning content that serves the strategies and priorities of theorganization.

Referring again to FIG. 2A, the business strategy alignment process 200initially receives a request in step S215 for learning services (a“learning effort request”) from an individual or business unit of theorganization. The learning effort request details what the individual orbusiness unit submitting the request believes is an important learningneed that requires a new or modified learning effort. According toembodiments of the invention, a learning effort request can be made inany form, including a verbal communication, a voicemail message, aletter, or an email message directed to a contact for the businessstrategy alignment module. Preferably, the request is provided on astandardized form (which, even more preferably, can be electronicallysubmitted) that is completed by the requesting entity, and where theform contains queries designed to elicit sufficiently detailedinformation that allows the business strategy alignment module contactto understand the request sufficiently. After receiving a learningeffort request, in step S220, an initial determination is made regardingwhether the request is addressable through training efforts or whetherit would alternatively be best addressed through more simple measureslike documentation or a newsletter produced by the resources of therequesting business unit of the organization. In the event that thelearning effort request is not deemed suitable for the learningsolution, at step S225 a rejection is sent to the requester.Alternatively, at S225, a query for a more detailed learning effortrequest is sent to the requester.

Whenever a learning effort request is deemed to present a learning needthat is addressable by the learning solution, step S228 is performed.Step S228 determines whether the data that is available is suitable formeasuring performance of the learning solution. Step S228 includestransforming the identified learning needs into performance/impactobjectives that can be measured by well-defined performance metrics.Further, the data sources of the organization are examined to determineif they are suitable for measuring the performance metrics. Theperformance metrics may be determined with reference to relevant andavailable data sources. For example, if the organization wishes to makeimprovements in employee morale, it likely does not track data measuringmorale. However, if the organization desired a 5% improvement inemployee attendance, daily attendance data could be suitable to measuresuch a performance metric. In the event that the available data sourcesare not deemed suitable or available for the desired performancemetrics, at step S225 a rejection is sent to the requester, and therequester may engage other options to establish methods for obtainingsuitable data.

If suitable data is available to measure the impact of the learningsolution on business results of the requested learning effort, in stepS230 an initial action proposal for the study design and the preliminarylearning effort plan (“preliminary LEP”) are generated. The preliminaryLEP makes a pre-production judgment as to the type of delivery that willbe most advantageous (Web-based course for self-paced instruction, acourse involving live or virtual classroom time with an instructor,etc.) given the learning effort request and likely logistical/budgetconstraints, and whether the request should be addressed by creatingwholly new learning content or by modifying the content of an existinglearning effort.

In certain preferred embodiments of the invention, the learning solutionadopts a learning effort cost schedule which provides a standard set ofprices or charges that will be “charged” to the requesting entity foreach type of content development or delivery type that can be utilizedin preparing a learning effort in response to a learning request. Forexample, a first rate could be charged for each hour of high-fidelityweb-based training content developed, a second rate could be charged foreach hour of low-fidelity computerized content developed, and a thirdrate could be charged for each hour of instructor-led training actuallydelivered. Therefore, when a new learning request is received, theinitial action proposal as detailed in the study design and preliminaryLEP gets mapped at step S240 into one of the tracts/initiatives in thelearning blueprint 211 as appropriate. Alternatively, of course, thecost that will ultimately be charged to the requesting entity can beestimated in a custom manner on a case-by-case basis when necessary orpreferred. Once a price or charge has been estimated for the initialaction proposal, that estimated price is compared against the budgetrestraints for the tract and/or initiative to which the effort ismapped. FIG. 2C and FIG. 2D depict in exemplary fashion the mapping oflearning effort plans 214 onto various tracts 212 and initiatives 213 oflearning blueprint 211 according to certain preferred embodiments of thepresent invention.

At S250, a validation check is made to assess whether the preliminaryLEP is feasible given budget priority restraints imposed by the currentlearning blueprint. If it is not, process 200 returns to step S240 andrevises its initial action proposal to produce a modified (e.g., scaleddown or re-targeted within the learning blueprint) study design documentand preliminary LEP. This modified study design document and preliminaryLEP are then mapped to the learning blueprint (as before, again at stepS240) using a revised cost estimate, and the validation check isrepeated (at step S250). This process loop repeats itself until thevalidation check is successful, which indicates that the action proposalfits within business goals and priorities (as evidenced by, among otherthings, budget allocations associated with the learning blueprint). Atthis time, a finalized learning effort proposal (“LEP”) is prepared instep S260 for use by the content management module. This LEP containsmore detailed budget allocation information, delivery timelines, andinitial content design requirements for use by the other modules of thelearning solution.

It should be understood that in performing some of the processes of thebusiness strategy and measurement alignment module 110 it may benecessary for person to person interaction to occur between learningadministrators and executives or managers of the learner organization.For example, in preparing and revising a learning blueprint and inestablishing budget allocations for learning initiatives it may benecessary for the business strategy and measurement alignment module 110to interface with the learner organization in order to identify andprioritize business strategies. In certain embodiments of the invention,such an interface can be provided by a learning management councilincluded of representatives from the learner organization. Optimally,the representatives are taken from all major segments of theorganization's business, such as various product segments, businessgroups, administrative groups (e.g., human resources and informationtechnology), and regional/international divisions and channels, toprovide a comprehensive interface with the ever-changing goals of theorganization. This learning management council could thereby serve as anintermediary between learning administrators and strategy-driving uppermanagement of the learner organization.

The learning solution 100 according to some embodiments of the presentinvention can facilitate the dissemination of critical learninginformation by internally developing deliverable learning content thatis consistent with the business needs of a specific organization. Byaligning the development of the information with an organization'sperformance goals according to its own performance data, the learningsolution 100 presents the critical information to the students in alogical, intuitive and valuable manner that is designed to enhancestudent receptivity and knowledge retention.

FIG. 3 illustrates a process 600 for using an organization's existingdata sources to measure performance of a learning solution. In stepS610, the business strategy alignment module instructs the datacollection module to set up interfaces to the organization's existingdata sources in order to collect data, and create reports for collecteddata. In step S620, a mechanism is chosen to isolate the businesseffects of the learning solution. Possible mechanisms include the use ofa control group, the use of historical data to create a baseline, andthe use of estimated data to create a baseline. The mechanism is chosenaccording to the organization's data infrastructure and needs. Forexample, the use of a control group requires data that is granularenough to separate results for members of the control group from others.Similarly, in order to create a historical baseline, actual historicaldata must be available.

In step S630, data is collected and reported by the data collectionmodule to the business strategy alignment module. A report is used incombination with the business effect isolation mechanism to determinethe change in the performance metrics in the learning blueprint. In stepS640, this change is converted to monetary value. For example, a 5%increase in sales volume might translate into $100,000 in increasedprofits. The exact computation to use is determined by business rulesfor the particular performance metric. Step S650 uses known techniquesto combine the change in monetary value with the cost of the learningsolution to compute the return on investment (ROI) of the learningsolution.

FIG. 4 shows a content development process 300 that embodies acombination of iterative steps that enable a learning solution toleverage benefits of the business strategy alignment module processesand thereby identify, develop, and manage deliverable learning contentinternally for an organization's required learning efforts. Contentdevelopment process 300 starts by interfacing in step S310 with businessstrategy alignment processes to receive information regarding thelearning content that needs to be developed for a learning effort (suchas by receiving a LEP produced according to the process 200 of FIG. 2Aabove). This passage of business strategy alignment information helps toensure that any new or modified learning materials and content arealways aligned with the business learning strategies and objectives of alicensing organization. The business strategy alignment information,such as in the form of a LEP, is then passed down (and possibly revisedas described below) through the various subsequent steps of contentdevelopment process 300, including content design step S320, contentproduction step S330, certification step S340, and delivery preparationstep S350 until it is ultimately handed off with any newly-developeddeliverable learning content at step S360 by interfacing with theprocesses of the delivery module.

In operation, upon receiving the initial content design requirementsfrom the business strategy alignment processes (such as in the form of aLEP), the content design step S320 leverages technology and organizationservices to implement a series of functions, tests, and analyses toproduce an initial layout of any new content that needs to be developedfor a given learning effort. The functions, tests, and analysesperformed during content design S320 include performance strategy androle impact strategy to make an initial assessment regarding arecommended instructional design (i.e., whether to utilize computerdelivered self-paced courses, instructor-led training (“ILT”), virtualclassrooms, etc.). In making this initial assessment, care is taken toperform audience analyses and environment analyses to ensure that thechosen instruction design is not incompatible with the logistics of thelearning audience (e.g., whether it is feasible to expect the learnersto be able to attend pre-scheduled live classes with an instructor) orwith learning services resources (such as instructor/classroomavailability, budget and development schedule constraints). The initialassessment will often also include media layout design (i.e., for acomputer-delivered interactive course, this would include a layout ofthe types of audio, video, and graphical presentations and simulationsthat would be required or useful in teaching the subject matter) andindications as to whether subject matter experts (“SMEs”) would benecessary for consultation.

In the event that additional information is necessary or that thelearning content cannot be designed within budget or logisticalconstraints, feedback can be provided upstream to the business strategyalignment processes to obtain revised initial content designrequirements (such as in the form of a modified LEP). This feedback isindicated in FIG. 4 by the broken arrow going from step S320 to S310,with the broken nature of the arrows indicating the optional upstreamaspect of this flow. All of the information included in the initialdesign assessment is used to produce a development plan that issensitive to budget and logistical constraints and delivery timelinesdetailed in the LEP, with the development plan then being referenced bythe later steps of content development process 300.

The results and/or conclusions detailed in the development plan asproduced from the various analyses performed during content design stepS320 are then utilized for downstream content production step S330,certification step S340, and the delivery preparation step S350 so as toensure that the development of a course's deliverable learning contentand materials remains consistent with the organization's businesslearning strategies and objectives and particular learning effort need(as reflected in the LEP and development plan). During contentproduction step S330, technology and organization services are leveragedto develop, license and build suitable learning content for thecarefully designed course curricula. This content production step S330can include, for example, subject matter information collection andorganization via SMEs, web-based training (“WBT”) learning contentobject development for self-paced learning courses delivered viacomputer, ILT and virtual ILT (“v-ILT”) curriculum layout and schedulingdevelopment, multi-media learning content development (including bothpre-production and post-production development) for use in one or morecourses of various types, packaging and integration with pre-existinglearning content, negotiation of any licenses or rights to use purchasedcontent, instructor development and training, and job-aid production.

As with the prior content design step S320, results of the ongoingcontent production step S330 can be communicated upstream to the contentdesign step S320 to facilitate peer review and enable revisions tocurrent design plans or even further upstream to enable LEP revisions ifnecessary. Once learning content has been finalized (or, optionally, atrail version capable of being reviewed has been finalized), acertification step S340 is performed to review the output of the contentproduction step S330. This certification compares the produced contentagainst the design requirements of the LEP and development plan toensure that the overall course development, production, and packagingare consistent with the licensing organization's business learningstrategies and objectives before the content is certified and releasedfor delivery. Any content that does not match these design requirementsis sent back to the content production step S330 stage (or earlierstages) for revision as shown by the broken arrows.

After the design step S320 and production step S330 of course contentand materials followed by their certification step S340, the certifiednew or revised content is subjected to a delivery preparation step S350.For computer delivered instruction objects and the like, this deliverypreparation step can include formatting or versioning of any contentobjects to be compliant with industry standards of electronic learningcontent delivery (such as the SCORM standard), and tagging andcataloging of the content objects to simplify their storage, retrieval,and ultimate delivery to learners. Content development process 300 endsby interfacing in step S360 with the delivery module and providing thecompleted and properly formatted, versioned, tagged and catalogeddeliverable learning content objects for learning use.

As will be readily appreciated by one of ordinary skill in the art, itwill not always be the best approach for the learning solution 100 tointernally develop deliverable learning content in response to alearning effort requested by the organization. Often, time constraints,subject matter constraints, and other logistical issues may make it morecost effective for the learning solution to purchase or licensepreviously developed learning content (completely ready for delivery tolearners or otherwise) or commission the development of suitablelearning content from third-party providers. For example, if theorganization needed employee training for a new computer system, it maybe more efficient to purchase training or reuse training materials fromthe provider of the new computer system as opposed to creating newlearning content internally from scratch. Therefore, the contentmanagement module 120 according to embodiments of the invention alsocontains processes that enable the procurement of third-party learningcontent and services as well as integrating that content and servicesfor delivery and administration within the control of the learningsolution.

As was the case with internally developed learning content, any learningcontent purchased or licensed from third-party providers would need tomeet the detailed budget allocation information, delivery timelines, andinitial content design requirements established by the business strategyalignment module (such as is detailed in a finalized LEP). Also similarto how was described above with respect to the content developmentprocess 300, any content obtained from third-party sources would becertified, formatted, versioned, tagged and cataloged in roughly thesame manner as internally developed learning content. After receivingthird-party content, the content management module still, like inprocess 300, certifies the content against requirements established bythe business alignment module. Any electronic learning content wouldthen be versioned and/or formatted to be compatible or consistent withinternally developed content (such as if, for example, digitalself-paced learning content materials need to be converted into SCORMstandard compliant format for later asynchronous electronic delivery tostudents). Such electronic learning content would then be taggedappropriately and placed into data stores to facilitate organization,delivery upon demand, and location/reuse of the content for the currentand future learning efforts. In this manner, content management module120 allows the learning solution 100 to take advantage of the mostefficient mechanisms for assembling appropriate learning content forefforts needed by the organization.

As indicated above with respect to the discussion of FIG. 1, themechanisms and media through which the delivery module 130 provideslearning content to learners of the organization is adapted to servicethe overall business strategies and specific learning efforts needs ofthe organization by organizing delivery to learners in the most resourceefficient and cost efficient manner. Suitable mechanisms and media bywhich learning content may be delivered to students according toembodiments of the present invention can include combinations ofcomputer-delivered or web-based courses available over electronicnetworks (such as the Internet) for self-paced instruction, interactivevirtual classroom instruction and related media (both video andanimation) available over electronic networks which utilize instructorsto lead classrooms of remotely located students, in person ILT, andpaper-based products used in conjunction with by self-paced orinstructor training

Turning now to FIG. 5, there is depicted the interaction of severaldelivery sub-modules which together interact to include a deliverymodule 130 of a learning solution 100 according to embodiments of thepresent invention. A delivery planning sub-module 410 and deliveryexecution sub-module 420 handle the coordination and actual delivery oflearning content, while delivery wrap-up sub-module 430 collects anddistributes data regarding completed learning content delivery. Thedelivery operations sub-module 440 concurrently serves as communicationinterface between the delivery module 130 and the other modules of thelearning solution. The delivery operations sub-module 440 also serves asa communication interface between the delivery module 130 andthird-party service providers to whom portions of learning efforts arecontracted, such as education service providers (“ESPs”) and value addeddistributors (“VADs”).

The delivery planning sub-module 410 contains various processes thatplan for upcoming new learning efforts by forecasting demand (sessions),scheduling delivery and resources timelines and allocations, andconfirming registration of learners prior to delivery execution. In someembodiments of the invention, coordination of such delivery planningprocesses are detailed in a delivery support plan (“DSP”). The DSP is atool or document that is created and grows throughout the deliveryprocess. For example, all of the information about a learning effortcourse that is being offered to the organization, including forecasteddemand and/or actual demand from for the course from the organization,is reflected in the DSP. The information and data within the DSP canthen be used to aid in future forecasting of demand for the same courseor for similar or related courses by the delivery planning sub-module410. The scheduling of resources and delivery coordination that occursduring delivery may include regional and global scheduling as necessaryfor a given learning effort. Additionally, in certain embodiments of theinvention, a scheduling synchronization tool may be used to facilitatethe management of scheduling and decision making information, includingthe identity of the instructors who are teaching each course, thelocation of where the course is being taught, the allocation ofmaterials and equipment that are being used by the instructor to teachthe course, and the usage of any electronic resources for virtualinstruction.

The delivery execution sub-module 420 is communicatively coupleddownstream from the delivery planning sub-module 410. This sub-modulecontains processes that coordinate the occurrence of field trials,training sessions for instructors in new course curricula, distributionof session materials (e.g., manually for instructor lead training (ILT)and printed materials, and electronically for virtual ILT andclassrooms, and Web-based or other self-paced computer-deliveredinstruction), and session set-up and breakdown processes. For example,the training for instructors can include helping them learn how best toteach certain content within changing course curricula by providinglists of recommended goals for the instructor to accomplish during thecourse.

As depicted in FIG. 5, the delivery wrap-up sub-module 430 iscommunicatively coupled downstream from the delivery executionsub-module 420. The delivery wrap-up sub-module 430 contains processeswhich close delivery execution sessions and generate delivery-relateddata for quality control checks and performance metrics for learningeffort and delivery success. Quality control checks can relate to themanagement of the performance and quality of instruction, includingsolely computer-delivered learning content and learning contentdelivered by both internal and third-party instructors. For example,instructors could be measured against pre-defined standards ofinstruction satisfaction by having students complete surveys orchecklists that are later reviewed and correlated with other performancemetrics (such as learning-output performance metrics like raw studentresults on certification exams administered during a course). Thedelivery wrap-up sub-module 430 optionally collects relevantdelivery-related performance metrics data for reporting via the deliveryoperations module 440, including, for example, student courseattendance, time it takes for an average student to complete aself-paced course unit, student scores on tests, instructor hours, etc.

As described above, the delivery operations sub-module 440 may serve asa communication interface between the delivery module 130 andthird-party providers, such as ESPs and VADs and as an interface withother modules of learning solution 100. Other processes that may becontained within this sub-module address delivery process improvement,financial management, resource management, facilities management,performance analysis, delivery support plan maintenance, instructorcertification, vendor/supplier management, remote learning servicescoordination, translation and localization of courses, archiving ofcourses, and business partners/VAD program administration. In anembodiment, the scheduling tool described above with regard to thedelivery planning sub-module 410, may also be used to help in resourcemanagement and facilities management by the delivery operationsub-module 440. Such resource management would include the management ofinstructors by determining their eligibility to teach a certaincurriculum and by cultivating their career growth while facilitiesmanagement would involve managing the buildings and equipment necessaryfor delivery of the curriculum. Further, instructor certificationprocesses can interface with processes for training instructors(described above and contained in the delivery execution sub-module 420)to make certain that internal instructors, as well as third-partyinstructors, meet minimum standards of effectiveness.

Referring back to FIG. 1, embodiments of the learning solution 100according to the present invention require administrative functionsupport which is provided by the learning administration module 140. Thelearning administration module 140 coordinates the process that respondto various administrative factor inputs as necessary to successfullyoperate and manage a global learning system. The administrativeprocesses according to the present invention allow, inter alia, studentsto enroll for courses, instructors to finalize “grades” or other indiciaof course passage, and learning administrators to request and reserveresources for specific classes. While similar administrative functionsare present in various common learning environments, such as intraditional colleges and universities, the processes employed in thesetraditional environments to perform these functions are not capable ofintegrating these functions into a learning solution that enablesalignment with business strategies by, among other things, generatingperformance metrics.

Specific learning administrative functions that may be provided by alearning administration module 140 in embodiments of the presentinvention include, but are not limited to, enrolling students incourses, wait listing students enrolled for coursed that are closed,providing documentation to training coordinators, creating new userprofiles, confirming enrollment in a specific course, adding a coursesession to the learning system and notifying the instructor by e-mail,developing and approving course catalogs and entering the approvedcatalogs into the learning system, adding and reserving class resources,sending students pre-course materials, adding and modifying courselocations, canceling instructor sessions and unbooking allocatedresources, deleting scheduled meetings, distributing notices of changesby sending e-mails to interested parties, updating student transcriptsand attendance information, administering on-line course and instructorevaluations, mailing course completion materials to students, andsending course completion or passage/certification records to humanresources for employee files.

The learning solutions compile, track, report and utilize performancemetrics to assist in ensuring that learning efforts provided by thesolution coincide with business strategies and priorities in all aspectsof learning including the identification and prioritization of learningneeds, the selection of learning approaches, and the development oflearning content, and the coordination and delivery of learninginstruction. The performance metrics use the same objective indicia usedby the organization to gauge success. Performance metrics can becollected, tracked compiled, and reported automatically by informationgathering and sorting tools, as described below with respect to FIG. 6.

As described above, embodiments of the learning solution providedaccording to the present invention are “outsourceable” in that they caneither be outsourced by the learner organization (whereby the processesof the learning solution are implemented by a third-party learningservices provider under contract to the learner organization) or can beimplemented internally by a sub-unit within the learner organization(as, for example, an independent learning business unit within the“outsourcing” organization). In embodiments of the invention wherein alearning services business unit within an organization provides learningservices to other business units within the organization, the learningservices business unit would still use the organization's existingperformance metrics to determine its success.

In some embodiments of the present invention the learning solution isprovided by an independent business entity according to a service levelagreement with an organization in need of learning services outsourcing.A service level agreement is a contract between the learning solutionservices provider and the licensing (i.e., outsourcing) organizationthat establishes the requisite quality and effectiveness levels of thelearning solution services as measured by one or more pre-selectedperformance metrics.

The terms of a service level agreement may require (as compared with thelevels measured before the implementation of a given learning effort),for example, minimum performance metric increases regarding gross salesgenerated by each trained salesperson. Such an outsourced learningsolution tracks and reports pre-selected performance metrics to theoutsourcing organization on a continuous basis to facilitate costallocation among appropriate business units or to specific learningefforts (equivalently, “training efforts”), and uses them within thelearning solution to gauge the level of effectiveness of particularlearning efforts in helping the organization reach desired businessobjectives and thus refine future learning efforts and overallstrategies.

Furthermore, the business management module provides the learningsolution with the capability to manage the overall lifecycle of learninginitiatives and efforts and to refine the processes employed toidentify, develop, manage, and deliver ongoing and future learninginitiatives and efforts in a “profit” driven manner. To achieve thisend, the business management module leverages selected performancemetrics on a regular basis, such as weekly, monthly, quarterly and/orannually, in various learning solution business management processes toensure that the learning solution is satisfying its “clients” andutilizing resources to do so in the most efficient (i.e., “profitable”)manner possible. For example, the business management module can compileweekly status reports and forecasts from selected benchmark performancemetrics to determine effectiveness of particular learning solutions anddetermine where remediation may be appropriate. Likewise, the businessmanagement module can collate and report performance metrics on amonthly basis as a comparison to commitments required by a relevantservice level agreement, and generate appropriate billing reportsregarding learning services rendered. Additionally, quarterly reportingof performance metrics can report longer-term performance of thelearning solution in comparison to minimum performance levels stipulatedin a service level agreement. Such quarterly reports can be used by thebusiness management module to forecast future fiscal year learningservices budgets for a learner organization or for making, as necessary,adjustments to the organization's current learning blueprint inconjunction with the business management module. Finally, yearly (orlonger term) reports also containing compilations of relevantperformance metrics data, indications regarding the level of success inmeeting service level agreement commitments, and financialcost/performance information can be used by the business managementmodule to generally re-assess the business strategies and prioritiesbeing pursued by the learning solution, including in negotiating outputlevel criteria for future service level agreements, in assisting thebusiness strategy interlock module in preparing new learning blueprintsfor upcoming fiscal years, and in preparing learning budgets and budgetallocations for the new blueprint.

The operation of the modules of learning solution 100 is preferablyfacilitated by various information sharing tools provided by a computingnetwork. FIG. 6 schematically depicts a network 500 of learningplatforms that communicate to provide coordination and synchronizationof learning content development, delivery and management so as toenhance the learning solution's response time to organization,instructor, student, and/or resource concerns and issues. The learningplatform network 500 according to such embodiments integrates variouselectronic tools to collect data from an organization in order to allowthe computation and use of business performance metrics. As shown inFIG. 6, a learning platform network 500 includes a content developmentplatform 510, a resources synchronization platform 520, an electronicdelivery platform 530, and a data collection platform 540 allcommunicating electronically with one another. Each of the platformsthat include the network 500 are composed of suitable servers, storagedevices (including databases), memory devices and support hardware as isknown in the art of computer networks to achieve the functions of eachplatform as described herein.

As shown in FIG. 6, the network 500 also interfaces with various users(including learning solution administrators) to facilitate thedissemination of learning resource allocation information, on anas-requested basis, and thereby enable informed decision makingregarding the utilization of various learning resources (personnel,electronic, physical, etc.). The resources synchronization platform 520of the learning solution network 500 collects and tracks resourceavailability information regarding deliverable learning content, anddelivery resources (classrooms, virtual classroom technology, targetlearner participation windows, instructor schedules) and organizes thisinformation into sortable scheduling and usage reports that allowlearning administrators to make and/or modify schedules and resourceallocations for learning content delivery.

The resources synchronization platform 520 further provides a meanswhereby performance metrics are compiled and tracked. These arecollected from the data collection platform 540. The resourcessynchronization platform 520 similarly provides various reports of theseperformance metrics to assist administrators in performing variousprocesses of the business management module as described herein,including the reconsideration and revision of business strategies andpriorities, associated learning budget allocations, and service levelagreements.

Content development platform 510 of network 500 automates many functionsassociated with the content management module 120 by providing contentauthoring and compiling tools. As shown in FIG. 6, the contentdevelopment module provides user interfaces for developing new ormodified learning content, accepting learning content information (suchas from external third-party sources), certifying, versioning andformatting new learning content data to comply with downstream networkrequirements (such as formatting self-paced computer-delivered learningcontent objects to comply with SCORM standards), and tagging andcataloging new or modified learning content for storage in a learningcontent data warehouse. Content development platform 510 also providesaccess to electronic authoring tools for formatting and compilinglearning content information into computer-deliverable electroniclearning objects.

Electronic delivery platform 530, as depicted in FIG. 6, providesnetwork 500 with its primary interface with the target learners (i.e.,the employees of the learner organization). The electronic deliveryplatform 530 has tools that enable asynchronous delivery of computerdelivered instructional content (preferably remotely over a securenetwork, such as via a web-browser over a secure WAN) to authenticatedtarget learners and track testing and/or certification results fromexaminations administered in conjunction with electronically deliveredinstruction. Additionally, platform 530 also provides tools that enabletarget learners to participate remotely in live classroom instructionvia virtual classroom technologies.

In use, learning solution network 500 facilitates the various processesas described above that embody learning solutions according to thepresent invention. By way of illustration, once a new action plancalling for new or modified computer-deliverable self-paced instructioncontent has been sufficiently finalized in response to a learning effortproposal (such as in the form of the LEP prepared for and provided tothe content management module 120), content developers can utilize thevarious electronic content authoring and compilation tools of thecontent development platform 510 to author electronic learning contentobjects. Such tools, among other things, enable the creation andorganization of object-oriented asynchronous computerized learningcourses (henceforth, “OOAC courses”) as well as the development of theirconstituent electronic learning content objects. The tools provided bythe content development platform 510 also enable importing ofthird-party information, content, and objects for incorporation intoOOAC new or revised courses. Such electronic learning information andin-development learning content objects are stored in a developingcontent database 510 a until they are ready (i.e., compiled, formatted,versioned, tagged and cataloged) for electronic delivery to students.The completed content objects are then electronically communicated tothe resources synchronization platform 520 for storage and subsequentdelivery.

Resources synchronization platform 520 accepts new or modified learningcontent from the content development platform 510 and stores it in acontent warehouse database 520 a (along with pre-existing contentobjects) according to appropriate tags and catalog identifiers assignedby the content development platform 510. These tags and identifiersallow the network to locate and identify content objects relating toparticular courses and deliver the appropriate objects upon subsequentdemands initiated via the electronic delivery platform 530. Thesynchronization platform receives electronic requests for learningcontent from the electronic delivery platform 530 and initiatesprocessing of these requests by retrieving the requested data from thecontent warehouse database 520 a.

On an ongoing basis, the resources synchronization platform 520simultaneously provides an interface for learning administrators toreview the current allocation of learning resources (as detailed in aresource schedule database 520 b) through customized scheduling andusage reports, and then use these reports for making resource allocationdecisions (using a graphical interface of a scheduling tool provided bythe resources synchronization platform 520) as appropriate based uponsystem constraints, current allocations, demands by learning efforts forlearning solution resources, and the relative priorities of thesecompeting demands for resources. Any resource allocation decisions madein this manner are appropriately reflected in the resource scheduledatabase 520 b. The allocation data stored in the resource scheduledatabase 520 b, and thus the reports provided by the resourcessynchronization platform 520, can include the current allocations andavailabilities of resources used by all learning efforts of the learningsolution 100, including classrooms, instructors, virtual classroomtechnology, OOAC delivery technology, as well as include deliveryconstraint information relating to target learner participation windowsand organization-imposed delivery deadlines. In this manner, decisionmaking regarding the scheduled delivery of learning content (bothelectronic and otherwise) can be synchronized across an entireorganization in a proactive manner to avoid schedule conflicts andinefficient usage of finite learning resources.

Simultaneously, the resources synchronization platform 520 preferablyprovides means for accessing correlated performance metrics data that isautomatically collected and stored for purposes of enabling the businessmanagement of the learning solution. These performance metrics arereported by the data collection module 540. A performance metricstracking and reporting tool of the resources synchronizationadministration platform 520 provides electronic access for learningadministrators and executives to the performance metrics database 520 cand thereby enables them to generate up-to-date reports regardingperformance metrics they select at any time and for any time period. Inthis manner, the performance metrics data that determines the return oninvestment for a learning solution can be centrally collected and moreeasily utilized, which may be useful, for example, where minimumperformance levels are dictated in a service level agreement or where alearning solution may be modified upon review of performance reports.Understandably, reports detailing such performance metrics data can beused by the business management module and business strategy alignmentmodules as described above to assemble, reassess and revise learningblueprints, learning solution budgets and overall strategies andpriorities as necessary.

Electronic delivery platform 530, as shown in FIG. 6, contains adelivery management database 530 a. The delivery management database 530a contains catalog and learner access information relating to coursesand related learning content so as to control access by various targetlearners (i.e., employees) to appropriately authorized learning content.This delivery management information is used by a learner deliveryinterface to provide virtual classroom data (as received from livevirtual content feeds) and OOAC course data electronically toauthenticated student users upon demand.

The data collection platform 540 interfaces with an outside data source.The organization's data source sends performance metrics to the datacollection platform 540, where it is stored in performance database 540a. The performance database 540 a includes fields for name, description,units, target, owner, trend, period, comment, etc. as required by theorganization data source. The data collection platform 540 contains astatistical package to allow functions including reporting trends onmetrics, identification of dependent and independent variables, andmodeling relationships among variables. The data collection platform 540further contains rules to allow it to translate business events (i.e.,changes in metrics over time) into monetary value. The data collectionplatform 540 also includes logic to break down a learning solution intoindividual learning activities and identify trends in metrics caused byindividual learning activities. The data collection platform 540 furtherincludes the ability to compare metrics against historical data.

1. A method for aligning employee learning efforts with strategicbusiness goals and priorities of an employer organization, said methodcomprising: receiving a set of business goals from the employerorganization; identifying performance data collected by the employerorganization; transforming the business goals of the employerorganization into performance metrics relative to the performance data;designating business tracts and initiatives for use in sorting potentiallearning efforts of the employer organization, the tracts andinitiatives correlating to the performance metrics; receiving requestsfrom the employer organization for learning efforts to train theemployees and sorting the requests into tracts or initiatives; selectingappropriate learning effort approaches for each received request, theappropriate learning effort approaches being subject to a budgetallocation for the tract or initiative into which each learning effortrequest is sorted; monitoring the performance data subsequent todelivery of the learning effort and computing changes in the performancedata caused by delivery of the learning effort; reporting measurement ofthe performance metrics based on the computed changes in the performancedata.
 2. The method of claim 1, further comprising analyzing the set ofbusiness goals and the collected performance data to determine thefeasibility of using the collected performance data to measureachievement of the set of business goals.
 3. The method of claim 1, thestep of transforming the business goals of the employer organizationinto performance metrics relative to the performance data furthercomprising identifying performance gaps, where the performance gap isthe difference between a current value of performance data and a desiredvalue of performance data.
 4. The method of claim 1, the step oftransforming the business goals of the employer organization intoperformance metrics relative to the performance data further comprisingidentifying sources of data needed to measure the performance metrics,the sources of data being provided by the employer organization.
 5. Themethod of claim 4, further comprising determining an appropriate datacollection method for the identified sources of data.
 6. The method ofclaim 1, further comprising using an isolation mechanism to isolate theeffect of the learning effort on the performance data.
 7. The method ofclaim 6, the isolation mechanism being one of: use of a control group,use of historical data to create a baseline, and use of estimated datato create a baseline.
 8. The method of claim 1, the step of reportingmeasurement of the performance metrics further comprising converting thecomputed changes in the performance data to monetary values.
 9. Themethod of claim 1, further comprising recommending remedial actionsbased on the reported measurement of performance metrics.
 10. Anetwork-based system to provide coordination and synchronization oflearning content development, delivery, and management, comprising: acontent development module, the content development module includingsubmodules for receiving input related to learning content andgenerating computer-deliverable learning content; an electronic deliverymodule, the electronic delivery module including submodules fordelivering instruction to students, the delivering of instructionincluding asynchronous computer-delivered instruction and synchronousvirtual classroom instruction, the computer-delivered instructionincorporating the learning content; a data collection module, the datacollection module including submodules for receiving performance datafrom an external data source and measuring the performance data againststored performance metrics, the performance metrics measuring theresults of the delivery of the instruction to students; a resourcessynchronization module, the resources synchronization module includingsubmodules for scheduling and allocating the utilization of learningresources in the delivery of the instruction and in the development ofthe learning content.
 11. The system of claim 10, the data collectionmodule further including a database for storing performance data forstudents.
 12. The system of claim 11, the data collection module storingperformance data for students in the database so that changes inperformance data over time are tracked.
 13. The system of claim 12, thedata collection module further including a submodule for statisticallyanalyzing changes in performance data over time.
 14. The system of claim13, the data collection module further including a submodule fortranslating statistical measurements of changes in performance data overtime into monetary values.
 15. The system of claim 10, the datacollection module further including stored historical performance data,the difference between the historical performance data and the receivedperformance data being computed.
 16. The system of one of claims 10-15,the data collection module further including a submodule for reportingmeasurements of performance metrics.